Evaluation of Endogenous Systems

In system development we are faced with the necessity of evaluation. Evaluation measures our success relative to other: a) theories of development or domains; b) implementations of similar theoretical principles; or c) increments of a given system. For successful software engineering evaluation, progress is measured against a model, a task frequently accomplished through requirements analysis. This model is lacking in natural language processing (NLP) systems (not to be confused with neurolinguistic programming). NLP systems defy evaluation in part because they model an endogenous process – where the whole process is irreducible. Therefore, while specific feature-based evaluations appear reasonable, they fail to capture an overall measure of success. In this paper, we look at the part/whole aspects of evaluation in more detail with regard to one language system type – machine translation. INTRODUCTION In system development, of any type of system, we are faced with the necessary evil of evaluation. Formal evaluation measures the success we are having relative to: a) other theories of development or domain modeling; b) other implementations of the same theoretical principles; c) other systems for the purpose of purchase or funding; or d) previous increments of a given system. For successful evaluation in software engineering, one needs a model against which progress can be measured, a task frequently accomplished through requirements analysis. For many domains, this is a straight-forward process: a correct answer exists, such as, “Pushing F10 results in program termination.” Many types of NLP systems, however, have defied rigorous evaluation because of this lack of defining criteria or requirements. Part of the difficulty lies in the fact that NLP systems are modeling an endogenous process – where the whole of the process is irreducible, context-dependent and lacking a unique right answer. Therefore, while specific feature-based evaluations appear reasonable, they fail to capture an overall measure of success. In this paper, we look at the part/whole aspects of evaluation in more detail with regard to one language system type – machine translation (MT). This paper starts with a description of the MT process as traditionally developed. It then describes the resulting evaluation strategies that follow from these views of MT. The features of MT which categorize it as an endogenous are presented as a precursor to showing a new view of the original MT vision. Finally, the questions of evaluation are explored within this new view.